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COST ESTIMATION: AN EXPERT-OPINION APPROACH 


Charles Buffalano 
Stephen Fogleman 
Mark Gielecki 

Goddard Space Flight Center 


INTRODUCTION 

At present three methods are generally used for cost estimating. The “engineering” or 
“grass-roots” estimating technique makes use of a work-breakdown structure— a hierarchical 
tree of work elements which comprise the item to be estimated. The cost of hardware and 
associated labor for each element is estimated and summed to yield total cost. The 
estimating by analogy approach directly compares the cost of a new item to the cost of a 
similar item that has already been buUt and for which cost data are available. The developed 
item becomes the baseline; that is, the one against which analogies are drawn and judgments 
about cost are extrapolated. Cost modeling is a statistical procedure that requires a large 
data base of many similar items. Cost is defined as being dependent on the technical char- 
acteristics of the item to be estimated. This relationship is described by a mathematical 
equation, called a cost-estimating relationship, and is obtained by regression analysis which 
employs a method of ordinary least squares on the data base. The appropriate parameters 
of the proposed item are then simply substituted into the equation, the equation is solved, 
and a cost is obtained. 

Although each one of these techniques has certain advantages, they are inadequate for 
estimating the cost of unique items or those for which old ways of doing business no longer 
apply. It is in these situations that optimum benefits can be derived from the use of the 
Quantified Expert-Opinion approach. 

DEFINITION 

The Quantified Expert-Opinion approach is a combination of two methods—the previously 
defined analogy estimating and the Delphi process of polling expert opinion. The Delphi 
technique is a systematic attempt to best utilize group judgment in areas where knowledge 
is incomplete. The basic premises of this method are that “many heads are better than one” 
and that a group will be more objective if there is no face-to-face confrontation. A typical 
Delphi process is run in the following manner. Each participant receives and completes a 
questionnaire. The identities of the other members are kept from individual members, or. 



as a minimum, the responses are kept anonymous. A control group then analyzes the answers 
to the questionnaire and forwards the statistical results of the first trial to the individual res- 
pondents so that they can compare their answers to those of the others. The participants 
at the extremes are asked to write short explanations of their positions, and these are relayed 
back to the group unsigned. Each participant then has the opportunity to change his or 
her position. This process is repeated until opinion stabilizes or, if this proves to be imprac- 
tical, for a predetermined number of times. Finally, the responses are statistically analyzed 
and aggregated to yield a group response. Experimentation with the Delphi method has 
shown that, in areas of partial information, it is superior to other methods of obtaining group 
response. For example, the Delphi approach minimizes the biasing effects inherent in group 
discussion. In particular it reduces the undue influence of dominant group members, dis- 
cussion not pertinent to the specific problem, and pressure to conform to majority opinion. 

For cost estimating, the Delphi method is used in conjunction with the analogy -estimating 
technique; that is, the expert opinions are made relative to an item which has been chosen 
to serve as a baseline. This integration of the analogy and Delphi processes is referred to 
as the Quantified Expert-Opinion approach to cost estimating. 

METHODOLOGY 

The Quantified Expert-Opinion approach can be used to estimate the cost of a variety of 
projects and operations; for example, instruments, spacecraft, or ground operations. Because 
this approach was first used to estimate the cost of an instrument, instruments will be used 
as examples throughout this document. The necessary ingredients for the implementation 
of the Quantified Expert-Opinion approach are: (1) a baseline instrument for which cost 
data are available, and (2) the availability and cooperation of experts who are familiar 
with both the baseline and the new instrument. 

The Delphi panel members receive a set of questionnaires, one for each program phase. The 
scope of the questionnaires should be structured to permit the respondent to make a 
meaningful judgment. Figure 1 shows a typical instrument questionnaire. The participants 
assess the relative effort by subsystem and program phase for the new instrument as compared 
to the basehne. In addition to the relative assessment, each respondent also bounds his or 
her best guess by a high and low assessment for each subsystem. The members return the 
questionnaires, and the control group analyzes them. 

The analysis of the responses can take any of a number of forms. A simple method of 
showing the responses visually is to graph them as in figure 2. Each triangle (probability 
density function) mathematically represents one respondent’s assessment. The peak of each 
function is the respondent’s best guess, and the points of intersection with the horizontal 
axis are his high and low bounds. This type of display permits easy identification of extreme 
opinions. The respondents at the extremes are asked to write short explanations of their 
positions, and the explanations and the graph are relayed back to the panel members. The 
panel members now have an opportunity to change their opinions, and the previously 
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Figure 1. Typical instrument questionnaire. 
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Figure 2. Graph of questionnaire responses. 
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described process is repeated. This iterative process continues until opinion-changing stabili- 
zes or, if this proves to be impractical, for a predetermined number of times. 

The purpose of this analysis is to obtain an average cost and spread for the instrument based 
on the opinions of the experts. When the final iteration of the Delphi process is completed, 
the effort-assessment ratios must be translated into cost ratios. This can be accomplished 
by assuming a one-to-one relationship between the two. In figure 2, note that most of the 
respondents are grouped around a ratio of 1 . 1 0 which means that they think that, at the 
design and development stage, the instrument’s scan mechanism will require 10 percent 
more effort than that required for the baseline instrument. Assuming that a one-to-one 
relationship exists between effort and cost, the increased effort translates into 10 percent 
added cost, and, because the cost data for the baseline instrument are available, the estimated 
dollar amount is easily obtained. Repeating this procedure and summing over all subsystems 
and program phases yields the mean total instrument cost estimate. To derive the confi- 
dence intervals associated with the mean estimate, further analysis is required. The most 
convenient case is to assume that the relative effort-assessment ratios are normally distri- 
buted and that the subsystems are independent (Appendix A). Utilizing these assumptions, 
the mean cost for each subsystem could be added to yield a mean cost of the total system, 
and likewise for the variance associated with each subsystem. The square root of the total 
variance results in the standard deviation for the total program cost and permits a probabi- 
listic statement concerning the cost estimate. For example, there wUl be a 68.3-percent 
probability that the cost of the instrument being estimated will be within one standard 
deviation of the mean. 

Other forms of analysis are dictated when the normality assumption is deemed unreasonable. 
One approach would be to fit a functional distribution form to the data; another would be 
to run a Monte Carlo simulation (Appendix B). 

SUBSYSTEM SENSITIVITY ANALYSIS 

Besides yielding cost estimates, the data obtained from using the Quantified Expert-Opinion 
approach can also be used for subsystem sensitivity analyses. That is, the data may indicate 
where additional cost-analysis effort could be most effectively used. The concept is simple: 
expend additional effort where the uncertainties produce the largest cost spread. For ex- 
ample, consider two subsystems, one at 10 million dollars and the other at 5 million. If 
the experts are closely grouped on the first subsystem so that their spread is from 9 to 
1 1 milhon dollars, but widely spread on the second, from 1 to 9 million, it is more im- 
portant to have the experts concentrate their efforts on improving their estimate on the 
5-million-dollar system. In economic terms, the marginal utility of improvements in the 
estimate are higher in the 5-million-dollar system than in the 1 0-million-dollar system. 
Appendix C contains a detailed explanation of this analysis. 

Goddard Space Flight Center 

National Aeronautics and Space Administration 
Greenbelt, Maryland May 3, 1976 
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APPENDIX A 

NORMALITY AND INDEPENDENCE 


The data can be analyzed in the following manner using the given assumptions. 

First, the data are grouped by subsystem and program phase. 

Second, each respondent’s modal value (estimate of most confidence) is multiplied by the 
actual cost of the baseline instrument for the particular subsystem and program phase. For 
example, if a respondent assessed the design and development effort necessary for the pro- 
posed-instrument scan mechanism to be double the effort necessary for the baseline instru- 
ment at the same subsystem and program phase level, the actual cost of the baseline-instru- 
ment scan mechanism would be multiplied by 2. 

Third, the modal cost figures are assumed to be normally distributed with. 
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where x. . is the respondent’s modal cost estimate of the subsystem, x. is the mean 

subsystem cost estimate, and n is the number of respondents. 

Fourth, the subsystems are assumed to be independent. 

Finally, the assumptions that the modal cost figures are normally distributed in each sub- 
system and that the subsystems are independent permit us to add subsystem distributions. 
Consequently, the total cost of five subsystems is normally distributed with, 
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where x^ is the mean cost for the subsystem, x^ is the instrument cost estimate, a? is the 
variance for the subsystem, and o represents the error bands of the cost estimate. 
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APPENDIX B 

MONTE CARLO APPROACH 


A simulation technique can be used to generate a cost estimate from the relative-effort 
assessment responses. An empirical distribution of modes, by subsystem, can be derived 
by sorting them in ascending order and assuming the lowest value to be the lower bound and 
the highest value to be the upper bound of all responses provided by the individual cost- 
estimating participants. These values depict the situation in which the relative-effort assess- 
ment is thought to be “as high, but no higher than” and, conversely, “as low, but no lower 
than” the particular values assigned. The distribution function is assumed to be linear be- 
tween the observed relative-effort assessment ratios, and each of the subsystems are assumed 
to be mutually independent. 

With the help of a computer, the first step in the program is to empirically derive the distri- 
bution of each subsystem by program phase and to linearly fit the data. Next, a random 
number sequence between 0 and 1 picks a value on each respective subsystem distribution 
and computes the corresponding cost by multiplying the randomly selected ratio of each 
subsystem by the relevant cost factor and by summing the costs. This process is repeated 
a large number of times, and thereby generates one cumulative distribution per program 
phase. This process is repeated for the remaining program-phase distributions yielding a 
final, total, cumulative distribution, mean, variance, and standard deviation. Typical out- 
put resulting from the use of the Monte Carlo approach is displayed in figure B-1. 
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Figure B-1 . Estimated cost of the proposed instrument relative to the baseline. 
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APPENDIX C 

SUBSYSTEM SENSITIVITY ANALYSIS 


Variance and standard deviation are measures of dispersion. Variance is defined as 

n 

j n ^ ij y 

i= 1 

In this model, a? is the variance of the subsystem, x.^ is the i^* respondent’s modal cost 
estimate of the subsystem, x. is the mean subsystem cost estimate, and n is the number 
of respondents. Total variance for the system is simply the sum of subsystem variances: 

± 

j=i 

Standard deviation is defined as the square root of the variance. The standard deviation for 
the total system gives the cost-estimate error bands and is calculated as follows; 


o=^ 

By definition, the sum of the standard deviations is not equal to total standard deviation. 

It is therefore impossible to place an exact dollar amount on the change in the cost-estimate 
error band that is attributable to a reduction of the variance in any particular subsystem. 

However, we can measure the responsiveness of the error range to a subsystem change in 
variance. To show this, take the partial derivative of o with respect to a., 

0^)(2ap a. 



The second derivative, 
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shows that, as is reduced, the impact of the reduction on the error band becomes greater 
(do Ido. is increasing at an increasing rate). That is, efforts to reduce o^ become increasingly 
cost-effective. 

The marginal effectiveness approach can be used to prioritize the instrument subsystems as 
to where effort can best be used to reduce cost spread. 
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